magazine coverAfter the Deluge
Despite a torrent of interest in ROI, truly workable solutions are just beginning to emerge.
Norm Alster, CFO IT
October 15, 2002

One by one, they rose to make their pitches to the IT steering committee. As the day wore on, recalls consultant Doug Hubbard, business cases were presented for more than 20 IT projects. Each was framed in terms of the tremendous savings and benefits it would provide for the company, a giant midwestern nuclear-power utility. One skeptical attendee listened closely and entered a series of figures into a calculator. Toting up the promised benefits of each proposal, he announced, "If we signed off on all of these, we'd be able to cut staff by 110 percent."

Writ large, Hubbard's tale reflects the past decade's blind faith, massive investment, and sometimes bitter disappointment in information technology. A recent study by technology research firm Gartner concluded that 20 percent of the $2.7 trillion spent worldwide on IT in 2001 was wasted. Others have found little or no correlation between technology spending and corporate performance. "Let's face it. There's got to be some increased sobriety about the value that can be created from technology investments," notes Christopher Dallas-Feeney, vice president, financial services group, at Booz Allen Hamilton. "It was overblown on Y2K. The ERP [enterprise resource planning] era was a bit overstated, and CRM [customer relationship management] is following on its heels."

Little wonder, then, that most corporate buyers are searching for new ways to gauge the payback from IT investments. Eager to oblige, vendors and consultants have trotted out a variety of tools that purport to more precisely measure return on investment. The approaches range from self-service Web sites that cough up an ROI calculation based on two or three inputs all the way up to new software programs costing as much as $200,000. Indeed, the mad rush to ROI is beginning to look like a tour bus unloading blackjack players in Las Vegas: everybody's got a system.

The problem with so much ROI analysis to date is that it's done by the very parties that champion technology spending, from vendors to consultants to systems integrators and outsourcers. When ROI is done internally, it's often done by the department that seeks the funding. "There's a fundamental conflict of interest when the ROI analysis is conducted by proponents of the project," notes Hubbard.

This is not, of course, news to CFOs, or even CIOs. "We try not to justify things too much on soft benefits and wishing," says John W. Prosser Jr., senior vice president, finance and administration, at Jacobs Engineering Group Inc. in Pasadena, California. Allan Woods, vice chairman and CIO at Mellon Financial Corp. in Pittsburgh, goes him one better: "We would not calculate productivity [gains] if there were no headcount reduction." And Ray Seabrook, CFO at Ball Corp., a Denver-based packaging firm, voices the general discontent with soft benefits and the companies that champion them when he says, "Five years from now they're sitting there with all our money in their pockets and I'm still trying to figure out how to measure something like 'employee empowerment.'"

In the face of such attitudes, vendors and consultants have redoubled their efforts to put hard numbers on soft benefits. Hubbard, for example, has created Applied Information Economics (AIE), an ROI methodology he bills as being "truly scientific and theoretically sound." He offers ROI services through his own firm, Hubbard Decision Research, and also licenses the AIE methodology to other consultants.

Steven Hausheer, chief operating officer at Investrics in Chicago, which is building the precepts of AIE into an ROI analysis software program, explains how it works. The basic concept is that things that don't seem measurable actually are. Consider something intangible such as "better employee access to information." Based on input from groups of employees, the Investrics program would try to answer such questions as: Would better information access result in faster decisions? Would it produce better decisions on pricing? Would it produce faster decisions that actually close sales and produce more revenue? Questions can be industry-specific; for insurers, would getting an answer to a prospect within one hour increase the chance of a sale, and, if so, by what percentage?

Similarly, for an oft-invoked intangible such as employee empowerment, the software would use input from employees (gathered by asking questions that have a scaled response, such as 5 for very likely, 1 for not at all likely, and so on) that zeros in on, say, the time that managers spend on supervisory tasks. If a project has, as one benefit, a reduction in such requirements, the process will be able to assign a hard-dollar value to at least one aspect of employee empowerment.

Hubbard says a system that assigns a probability-weighted range of values acknowledges the inherent uncertainty of input regarding projected benefits. Rather than yielding a single projected value, AIE calculations yield a probability-weighted range of outcomes. This range is reflected in the final ROI assessment, which is essentially a collection of such outcomes. Hubbard might, for example, conclude that a client has a 40 percent chance of a 50 percent ROI on a new document-management system. But recognizing such risks as project cancellation and the possibility that users might not fully embrace the system, he might conclude that there is also a 10 percent chance that the project will produce a negative return. Based on input from managers, Hubbard calculates the level of risk that a client will tolerate for a given projected return. If, in this case, managers have indicated they would tolerate as much as a 15 percent risk of negative return in exchange for a 50 percent ROI, the project would get a green light. If management will tolerate only a 5 percent negative return risk, that project would be rejected.

It is in the assessment of overall project risk that Hubbard and those who have drawn on his methodology distance themselves from vendors and some consultants who, to borrow from Will Rogers, have never met a tech project they didn't like. Hubbard says that his analysis results in a red light about 20 percent of the time; of the remaining projects, about 60 percent require modifications before getting a green light.

CFOs would do well to ask those peddling various ROI methodologies how often their analysis puts the kibosh on projects. With customers increasingly alert to the risks in new IT projects, some ROI consultants claim they have become tougher graders. "If two or three years ago 1 in every 2 or 1 in every 3 were green lights, now it's 1 in every 5 or 1 in every 10. The eye of the needle is getting narrower," notes Booz Allen Hamilton's Dallas-Feeney.

Then again, perhaps the devil is not so much in the details as in the share price. A company called iValue urges an approach to IT portfolio management that focuses on shareholder value. Co-founders Ray Trotta and Christopher Gardner argue that large IT projects should be assessed in the same way that Wall Street assesses companies, with the emphasis on how shareholder value will grow over time. Too often, they say, companies concoct some sort of ROI methodology up front in order to green-light a project with some degree of confidence, then let that project proceed as projects always have — with precious little focus on financial discipline.

They urge companies to develop simulations as a way to assess how and to what degree a project will yield a return over time. Pilot projects provide one form of input, as do market research, conjoint testing (in which customers or users of a system are interviewed about the trade-offs they're willing to make between cost and functionality), and other forms of analytics.

"ROI is just the latest management fire drill," says Gardner, whose book The Valuation of Information Technology (John Wiley & Sons, 2000) lays out much of the thinking behind iValue's methodology. "Companies need to change their entire approach to how they make decisions." It's ironic, he says, that Wall Street has imported computer science experts to bring technological sophistication to the value of companies, but IT departments have not tapped financial experts to help assess the true value of IT projects.

The sorts of changes that are needed don't come easily, and Trotta admits that one impediment to the adoption of iValue's approach is that "it takes work, and you have to follow through. Two complex disciplines, IT and finance, have to work together continuously." When they do, he says, the focus on ROI will give way to a more-continuous assessment of value creation.

Clearly what seemed at first blush like a panacea for the costs and complexities of IT projects — conduct an ROI analysis — is rapidly morphing into perhaps the central question for executives involved in IT strategy: how to create a solid framework in which IT spending can be analyzed and capital best deployed. There is certainly no shortage of firms willing to help you take some new swipe at these perpetually vexing questions. That means companies are in the unenviable position of having to analyze how they approach analysis. Yet those that meet this challenge successfully will be well positioned to stem the tide of millions of wasted dollars and untold hours of wasted effort. You may be tired of hearing about "ROI," but the real discussions are just beginning.

Norm Alster has worked for Forbes and Business Week and is currently a contributor to The New York Times.

Time Is Not Always Money

Improved productivity is one eternal promise of technology. But does the ability to produce more goods and services at the same cost actually translate into bottom-line gains? Often not, say many CFOs, who are wary of productivity gains that don't result in more goods produced or fewer people on the payroll.

Productivity gains don't necessarily boost revenue or cut cost, because people don't necessarily apply freed-up time to work — particularly in the executive suite. Ian Campbell, vice president of research at Nucleus Research Inc., says that, generally speaking, lower-level employees are more likely to turn extra time into more work. "It's the VP who says, 'I can close my door and practice putting,'" says Campbell. "If you're a line worker, there's no chance to goof off."

This reality, Campbell says, needs to be factored into ROI equations. Here are his rule-of-thumb calculations on the percent of freed-up time various workers convert into increased output: